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Introduction 


¢ All complex development projects experience delays and corresponding 
backlogs of their project control milestones during their acquisition lifecycles 


¢ NASA Goddard Space Flight Center (GSFC) Flight Projects Directorate 
(FPD) teamed with The Aerospace Corporation (Aerospace) to develop a 
collection of Early Warning Look Ahead metrics that would provide GSFC 
leadership with some independent indication of the programmatic health of 
GSFC flight projects 


¢ As part of the collection of Early Warning Look Ahead metrics, the Percent 
Milestone Backlog metric is particularly revealing, and has utility as a stand- 
alone execution performance monitoring tool 

¢ This paper describes the purpose, development methodology, and utility of 
the Percent Milestone Backlog metric 

¢ The other four Early Warning Look Ahead metrics are also briefly discussed 

¢ Finally, an example of the use of the Percent Milestone Backlog metric in 
providing actionable insight is described, along with examples of its 
potential use in other commodities 
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Early Warning Look Ahead Metrics Overview 


¢ Look Ahead Metrics provide a concise, graphical representation of the past, current, and 
potential future state of the programmatic health of a project 
— Has similarities to EVM, but is not the same thing — this looks at a project from a different 
viewpoint, and has a different underlying basis 
Examines milestone completion rate, milestone backlog, spending rate, and ability to finish 
all milestones on time 


Milestone Backlog 


Milestones to Go Milestones to Go (Enlarged) 


Cost 
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Early Warning Look Ahead Metrics Definitions 


¢ Milestones to Go Metric depicts the number of actual vs. planned milestones remaining until 
project completion 


— ls the project completing milestones in accordance with the plan? 
— ls the project on a path to complete all milestones in time to finish? 
¢ Milestone Backlog Metric depicts the actual milestone backlog each month 
— Is the milestone backlog growing or shrinking? 
— What is the backlog trajectory? Is the project on a path to recover? 


¢« Percent Milestone Backlog Metric enables direct backlog comparisons to other projects that 
have been successful 


— ls the milestone backlog something to be concerned about at this point in the project? 
— Have other projects had this level of milestone backlog and recovered? 

* Schedule Runway Metric depicts the number of months needed to complete all remaining 
milestones, given the recent milestone completion rate, and compares to actual amount of time 
remaining 

— Does the project have time to complete all remaining milestones? 
— Does the project need to increase its milestone completion rate? 


¢ Cost Runway Metric depicts the number of months of spending capacity remaining given recent 
spending rates, relative to the cost to go, and compares to actual amount of time remaining 


— ls the project spending money at an appropriate rate to finish at or under budget? 
— Does the project need to modulate its spending rate? 
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* Milestone Performance Metrics measure the | + Milestones-to-Go depict the number of 


degree to which project control milestones actual versus planned milestones remaining 
are backing up and being shifted to the right until project completion 
over time 


¢ Milestone Backlog depicts the actual 


— Aproject that is falling behind in milestone backup each month 


completion of its milestones will “push” 


those milestones out to the right, ¢ Schedule Runway Metric depicts the 
creating a “bow wave’ of milestones yet number of months needed to complete all 
to be completed milestones on time, given the recent 

— Impacts ability to finish project on time milestone completion rate 
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Milestones to Go 


Months to Go to Based on Milestone Completion 


A) AEROSPACE Percent Milestone Backlog Metric - 7 


: SSS 
FLIGHT PROJECTS DIRECTORATE 


Cost Performance Metric 


¢« Cost Runway Metric indicates whether or ¢ Cost Runway Metric is the ratio of the 
not a project has enough unexpended cost Cost-to-Go to the 3-month average spend 
remaining at its current spend rate to rate, and represents the number of 
make sure it is on a trajectory to finish months of spending capacity remaining 
before running out of money until available funds are expended under 


recent spend rates 


— If Milestones to Go Metric is less than 
the number of months remaining to 
enna Launch Readiness Date (LRD), then 
: : the project is in danger of running out 

of money before reaching LRD — the 
spend rate is too fast 
— Milestones to Go Metric w/Reserves 


is calculated using Cost-to-Go (plus 
reserves) 
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Example Combined Look Ahead 


¢ Combined Look-Ahead for EXAMPLE is mixed 


— Milestone completion rate has remained steady and is converging to the plan, meanwhile, 
the backlog is increasing and trending unfavorably, and exceeds historical limits for 
successful projects, but, nonetheless, sufficient time remains to complete all milestones 


— Spending rate is consistent with finishing on budget and trending favorably 


Milestones to Go Milestones to Go (Enlarged) Milestone Backlog 
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Wait! Isn’t This the Same as EVM? 


¢ No! Early Warning Look Ahead metrics are (mostly) independent of EVM 
— Some obvious similarities to EVM, but uses a different basis 
— Schedule basis is control milestone completion; reports milestones-to-go, 
milestone backlog, and months-to-go 
— Cost basis is monthly spending vs. development cost-to-go; reports months of 
spending capacity remaining 
¢ Most NASA flight projects utilize EVM to measure cost and schedule performance, 
but GSFC sought an additional, independent method of evaluating the programmatic 
health of its flight projects 
— Early Warning Look Ahead metrics complement EVM metrics 
— A “second opinion,” combined with other traditional EVM and performance 
metrics, to describe overall health of GSFC flight projects 
¢ But, not all projects utilize EVM. The Early Warning Look Ahead metrics can be 
constructed in lieu of EVM for projects that have chosen not to use EVM to monitor 
performance 
¢ The two types of metrics together provide a more complete picture of project 
performance than either type used alone 
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The Percent Milestone Backlog Metric 


¢ The Percent Milestone Backlog Metric provides actionable insight to GSFC 
management on the milestone execution performance of its flight projects 


— ls the project in a state of backlog that no other successful project has been in? 
— ls the backlog “normal” for the project at this point in time? 
— Is it time for management intervention? 
¢ Atool to monitor schedule and milestone completion performance 
— Compares to similar, successful, projects 
— Provides indication as to whether the backlog is “normal” or “in extremis” 


¢ Developed for GSFC Flight Projects Directorate, but could be adapted to a broader 
range of projects; with analogous performance data, could be applied to industry or 
other government agencies 


¢ This paper will discuss 
— Purpose 
— Development Methodology 
— Utility 
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Purpose of the Percent Milestone Backlog Metric 


¢ Itis not enough to compare a project’s milestone completion performance to 
its own plan 


— Every project has a different plan and a different number of milestones 
— How do we know if a project's backlog is “going off the rails?” 


¢ Should also be able to compare a project’s milestone completion 
performance to a collection of other successfully delivered projects. 
Have other projects had a backlog of this size and still delivered? 


¢ The Percent Milestone Backlog metric allows us to compare a project's 
milestone backlog to historical norms 
— lf the Percent Milestone Backlog falls within the “region of normal,” then 
there is no apparent cause for alarm 
— lf it has departed the “region of normal,” then it may be time for 
management intervention 
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Development Methodology 


Underlying metrics are the Milestone Backlog and the Cumulative Planned 
Milestones for a given month 


— The Percent Milestone Backlog for a given month is computed as 


Percent Milestone Backlog = 
Current Milestone Backlog/Current Cum Plan 


— This tells us the percent backlog of a given project in a given month 


— But, to develop a basis of comparison to other projects, normalization is 
required 
¢ The first step was to collect milestone performance data, both planned and actual, 
for a collection of historical flight projects that were successfully delivered, or close 
to delivery 


¢ Selected projects included: 


ASTRO-H SWIFT GPM LRO SPP OSIRIS-REx ICON 
Aquarius SAM MMS SDO ST-5 GOES-R 
LOLA SMAP IRIS MAVEN  RBSP GOLD 
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Development Methodology (2) 


¢ The projects’ milestone performance data were obtained from their respective project 
Monthly Status Reports (MSRs) 


— Anexample of a typical milestone status chart is shown below 


Cumulative Milestone Performance Status 
Status as of: Aug 2014 
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— From these charts, Percent Milestone Backlog metrics were computed for each 
month of the project’s development lifecycle from Preliminary Design Review 
(PDR) to LRD or delivery 
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Development Methodology (3) 


¢ The second step was to normalize each historical project’s schedule to a 0% (PDR) 
to 100% (launch or delivery) percentage schedule using a simple interpolation 
scheme. This enabled a consistent basis of time for each historical project 
¢ The third step was to calculate the percent milestone backlog of each historical 
project against the normalized schedule from PDR to launch or delivery 
— For example, suppose in September a flight project had a 38% milestone 
backlog at 17% of its schedule from PDR to launch or delivery, then in October 
the same flight project had a 43% milestone backlog at 19% of its schedule 
— We would interpolate the intervening 18% schedule as having a percent 
milestone backlog of 40.5% by interpolation as follows 


Percent Milestone Backlog 
Interpolated Value 


38% 40.5% 43% 
17% 18% 19% 


Sept Oct 
Percent Schedule from PDR 


— This process was repeated for each unit schedule percentage from 0% to 100% 
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Development Methodology (4) 


¢ The graphs shown below illustrate plotting the percent milestone backlog for one 
project, and also for the entire collection of projects 


Single Project Collection of Projects 


Percent Milestone Backlog 
EXAMPLE by Percent Schedule 
(Single Project) 


Percent Milestone Backlog 
EXAMPLE by Percent Schedule 
(Individual Projects) 
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Development Methodology (5) 


The fourth step was to define the region described by all of the historical projects as 
the “region of normality.” Then, for each percentage point along the schedule axis, 
the minimum, maximum, 50" percentile and 70" percentile percent backlog was 
computed in order to develop “shades of normality” 


Region of Normality Region of Normality divided into percentiles 


Percent Milestone Backlog Percent Milestone Backlog 
EXAMPLE by Percent Schedule EXAMPLE by Percent Schedule 
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Development Methodology (6) 


¢ The last step was to map the project being monitored against the region of normality 
This enables one to gauge the percent milestone backlog performance of the 
project relative to the collection of projects making up the region of normality 
This requires re-normalizing the region of normality to the schedule of the 
project being monitored as described previously (also done using an 
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interpolation scheme) 

Notice that the normalization back to 
the project’s schedule has resulted in 
some loss of granularity in the region 
of normality 

In this example, the percent milestone 
backlog for “this month” is about 10% 
and is projected to increase further 
However, the backlog is well within 
the region of normality, indicating that 
there is no real cause for alarm at 
this point 
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Interpretation 


¢ So, what is this chart telling us? 


The project being monitored has always had a positive backlog from PDR to 
the present 


— Looking at time = “now,” the backlog is at about 10% and projected to increase 


But, this is well within the norm for a project at this stage in its lifecycle, and there 

is currently little cause for alarm 

— The projection arrow exiting the 
current percent backlog is based on 
the 3-month average of percent 
backlog and indicates where we 
predict the backlog to go if the project 
continues at its current pace 

— The “cone of uncertainty” provides an | _* 
indication of milestone backlog :~ 
behavior over the past six months 

— Bottom line: Yes, there is a backlog, 

but we’re not too concerned about it 

at this point because many others 

have been there and recovered 
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Utility 
¢ The Utility of the Percent Milestone Backlog metric: 


Gives ability to compare a project of interest to a collection of similar projects 


Provides an “At-a-Glance” view of the project’s milestone completion 
performance 


¢ Past 
« Present 
¢ Future 
— Complements EVM as an indicator of project performance 
— Provides valuable historical context 
— Delivers independent indicator of programmatic health of a development effort 


— Could be applied to other commodities where performance is measured in 
discrete milestones 


¢ Industry 
¢ Other NASA Centers 
¢ Other government programs 
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Exampl 
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¢ Project XYZ has consistently been lagging in its milestone completion and 
has been building a backlog 


Milestones to Go Lagging the Plan 


Milestones to Go 
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Milestone Backlog has grown 


Milestone Backlog 


Backlog 
> Trend (3 mo avg) 


¢ It looks bad, but is Project XYZ really in trouble? 
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Example 


¢« Despite seemingly bad news, Project XYZ is actually in a better position 
than many past projects have been at this point in time 
Milestone Backlog has grown continuously Percent Milestone Backlog has remained steady 
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¢ The Percent Milestone Backlog metric indicates no cause for alarm...yet. 
Things are okay for now, but the projection indicates there may be trouble 
ahead if milestone completion performance doesn’t change 
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Conclusions and Recommendations 


¢ The Percent Milestone Backlog metric was developed for use by GSFC 
Flight Projects Directorate as one of several metrics to evaluate 
programmatic performance of its flight projects 


¢ The Percent Milestone Backlog metric provides actionable insight to 
management 

¢ The metric could be adapted to a broader range of projects but substantial 
historical data collection is necessary in order to be useful 

¢ Future work is needed to find more efficient methods for building up the 
databases and graphs needed to support the Percent Milestone Backlog 
metric 
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